Experimental Assessment of Automated Knowledge Capture

The U.S. armed services are widely adopting simulation-based training, largely to reduce costs associated with live training. Ideally, instructors would observe each individual within the context of team performance and provide instruction based on observed misunderstandings, inefficient task execution, ineffective or inappropriate actions and so forth. However, it is impossible for instructors to devote this level of attention and time to each student. To maximize training efficiency, new technologies are required that assist instructors in providing individually-relevant instruction. Sandia National Laboratories has shown the feasibility of automated performance assessment tools, such as the Sandia-developed Automated Expert Modeling and Student Evaluation (AEMASE) software, through proofof-concept demonstrations and a pilot study. In the pilot study, AEMASE achieved a high degree of agreement with a human grader (89%) in assessing tactical air engagement scenarios. The current study provides a more rigorous empirical evaluation of the accuracy of these assessments. We trained employees at Sandia National Laboratories on a Navy simulator and then assessed their simulation performance using both AEMASE and manual assessment. The participants were evaluated on four metrics including fleet protection, labeling of entities, management of Combat Air Patrol assets and radio chatter. Results show that AEMASE assessments were comparable to the manual assessments on at least three of the four metrics. The study showed that AEMASE surpassed target performance criteria with agreement of up to 100% with the manual assessments.